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INTRODUCTION 


Many  of  the  problems  in  developing  reliable  software  for  complex  computer 
systems  used  in  battlefield  automated  systems  are  related  to  defining  software 
requirements.  The  high  costs  of  software  development  and  poor  performance  grow 
from  ambiguous  software  requirements.  The  costs  are  directly  related  to  the 
difficulties  in  defining  software  requirements,  the  problems  with  coordinating  a 
large  programming  team,  the  difficulty  of  the  test  and  error  removal  process,  and 
the  subsequent  maintenance  and  enhancement  of  field  deployed  software. 

At  the  National  Conference  on  Software  Development  in  Chicago  which  was 
sponsored  by  the  Data  Processing  Management  Association  Educational  Foundation  in 
June  1983,  Alfred  Sorkowitz  reported  that  the  yearly  cost  of  software  developed 
in  the  United  States  was  estimated  to  be  over  20  billion  dollars.  (Other  esti¬ 
mates  from  recent  years  ranged  from  16  to  32  billion.)  Furthermore,  the  Govern¬ 
ment  Accounting  Office  studied  nine  federal  software  projects  and  determined  that 
only  approximately  2%  of  the  software  had  been  used  as  delivered  to  the  govern¬ 
ment  . 


The  Department  of  Defense  has  recognized  the  software  testing  problem,  and 
document  DoDD  5000.3  (Test  and  Evaluation)  was  established  primarily  to  enforce 
the  need  for  demonstrating  software  quality  (refs  1-3).  Special  emphasis  was 
placed  on  the  testing,  and  evaluating  of  software.  Within  DoDD  5000.3,  the  fol¬ 
lowing  four  points  are  worth  noting  since  they  strongly  relate  to  testing  in  the 
software  development  process: 

1.  quantitative  and  demonstratable  performance  objectives  shall  be  es¬ 
tablished  for  each  software  phase. 


2.  The  decision  to  proceed  to  the  next  phase  shall  be  based  on  quanti¬ 
tative  demonstration  of  adequate  software  performance  using  test  and  evaluation. 

3.  Prior  to  release  for  operational  use,  software  shall  be  operation¬ 
ally  tested  under  realistic  conditions  to  provide  a  valid  estimate  of  system 
effectiveness  and  suitability  in  the  operational  environment. 


4.  Operational  test  and  evaluation  agencies  shall  participate  in  early 
stages  of  software  planning  and  development  to  insure  adequate  consideration  of 
the  operational  environment  and  objectives. 


This  study,  directed  at  an  examination  of  test  tools,  was  encouraged  by  the 
recent  progress  on  these  tools  (refs  4-6).  The  objective  was  to  investigate  the 
development  of  standardized  methodologies  for  the  design  of  software  test  driv¬ 
ers  .* 


*  Test  drivers  are  software  programs  which  provide  data  for  exercising  and 
testing  software  that  has  been  completed  or  is  under  development. 


The  long-term  goal  of  this  project  is  to  develop  standard  methodologies  and 
test  drivers  for  the  Army  and  perhaps  other  Department  of  Defense  (DoD)  agencies 
in  order  to  assure  that  a  uniform  level  of  confidence  for  the  software  quality 
beyond  the  critical  function  stress  tests  is  achieved.  Stress  testing  is  related 
to  boundary  value  analysis  which  is  a  selection  technique  in  which  test  data  are 
chosen  to  lie  along  "boundaries"  or  extremes  of  input  domain  (or  output  range) 
classes,  data  structures,  and  other  procedure  parameters.  Choices  for  data  often 
Include  maximum,  minimum,  extraordinary,  and  degenerate  values  of  parameters.  As 
the  long-term  goal  becomes  achievable,  there  is  a  strong  probability  that  a 
higher  percentage  of  software  will  be  used  in  the  field  with  minor  error  correc¬ 
tions  within  the  software  development  life  cycle  process . 

The  initial  scope  of  work  for  this  research  project  was  prepared  and  sub¬ 
mitted  for  approval  in  the  last  quarter  of  FY82.  Briefly,  the  project  funding 
levels  were  revised  and  an  initial  funding  request  was  approved.  In  order  to 
complete  the  first  phase  of  the  project,  funds  were  approved  for  in-house  govern¬ 
ment  support  and  an  additional  amount  was  granted  for  outside  consulting  serv¬ 
ices  . 

The  first  phase  was  an  investigation  of  the  test  driver  literature  and  po¬ 
tential  automated  tools  to  explore,  while  the  second  phase  is  an  experimentation 
with  selected  tools  that  will  be  evaluated  for  effectiveness  in  the  software 
quality  assurance  functions  for  battlefield  automated  systems.  This  report  pre¬ 
sents  the  information  discovered  during  the  first  phase  of  the  research 
project.  Proposals  are  being  submitted  for  continued  funding  for  the  second 
phase . 


Software  Development  Life  Cycle 

Software  can  be  defined  as  the  computer  programs  and  data  required  to  enable 
computer  hardware  to  perform  computational  or  data  manipulation  functions.  Soft¬ 
ware  is  used  in  various  environments  including  battlefield  automated  systems  that 
are  being  investigated.  Such  software  may  include  operational  embedded  software 
which  is  used  for  system  performance,  built-in-test,  or  test  program  sets  which 
are  used  for  the  identification  of  malfunctions  in  the  software  and  maintenance 
activities . 

A  simplified  software  life  cycle  for  battlefield  automated  systems  can  be 
broken  down  into  the  following  stages: 

1 .  Conceptualization 

2.  Requirements  defined 

3.  High  level  design 

4.  Detailed  design 

5.  Implementation  of  design 

6.  Code  testing 

7.  Requirements  testing 

8.  Software  maintenance 


V 


Throughout  the  system  life  cycle,  there  are  various  phases  for  testing, 
verifying,  and  validating  the  software.  Test  drivers  can  be  used  in  these  vari¬ 
ous  cycles  or  phases  of  project  development.  For  example,  coding  or  requirements 
testing  is  an  ideal  phase  for  verifying  and  validating  software.  In  addition, 
the  software  maintenance  phase  is  critical  in  the  software  devel  ..ment  life 
cycle.  Test  drivers  are  necessary  in  software  quality  assurance  functions  in 
order  to  reverify  and  revalidate  software  changes.  They  are  used  in  regression 
testing  and  developing  test  beds  for  software  qualification  and  acceptance. 


Role  of  Testing 


Testing  is  described  by  individuals  in  different  ways.  For  example,  Webster 
uses  the  terms  "critical  examination,  observation,  or  evaluation"  when  he  defines 
testing.  Other  researchers  and  authors  described  testing  from  different  perspec¬ 
tives.  Glenford  Myers  (ref  7)  explains  testing  as  the  process  of  executing  a 
program  with  the  intent  of  finding  errors.  Michael  Deutsch  (ref  8)  perceives 
testing  as  the  controlled  exercise  of  program  code  in  order  to  expose  errors,  and 
Goodenough  defines  testing  as  a  process  inferring  behavioral  properties  of  a 
program  based  on  the  results  of  executing  the  program  in  a  known  environment  with 
selected  inputs. 

More  technially.  Miller  and  Howden  (ref  4)  explain  that  a  unit  test  of  a 
single  module  consists  of  a  collection  of  settings  for  the  input  space  of  the 
module  and  exactly  one  invocation  of  the  module.  A  unit  test  may  or  may  not 
include  the  effect  of  other  modules  which  are  invoked  by  the  module  undergoing 
testing. 

The  role  of  testing  in  a  DoD  environment  is  to  verify  and  validate  the  dem¬ 
onstration  of  computer  program  requirements  for  system  performance.  Verification 
can  be  defined  as  the  process  of  assuring  the  consistency  and  completeness  of  the 
current  phase  with  the  previous  phases  of  the  system  life  cycle  development  pro¬ 
cess.  Validation  is  the  assessment  of  the  final  software  product  through  test, 
evaluation,  and  demonstration  to  measure  how  well  the  software  conforms  to  the 
established  requirements. 

More  specifically,  the  role  of  testing  is  very  important  in  DoD  projects  and 
is  used  throughout  the  life  cycle  management  of  the  software.  Testing  ultimately 
is  used  in  formal  qualification  testing  and  acceptance.  This  later  phase  in  the 
life  cycle  process  is  used  as  a  prerequisite  for  final  acceptance  of  the  software 
technical  data  package  which  contractors  submit  to  DoD  agencies  for  approval. 

The  following  classification  schemes  for  test  techniques  are  discussed: 
performance  testing,  code  walkthroughs,  classification  of  errors  and  test  types, 
static  and  dynamic  testing,  set  theory,  input-output  space,  graph  theory,  and 
structured  testing. 
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Statement  of  Problem 


The  present  underlying  problem  In  software  development  In  DoD  agencies  Is 
that  much  of  the  software  developed  by  contractors  and  delivered  to  the  govern¬ 
ment  has  a  very  low  probability  of  being  used  as  delivered.  This  problem  can  be 
broken  down  Into  many  steps  and  subproblems  related  to  the  life  cycle  management 
process.  Clearly  defining '  software  requirements  early  In  the  project  Is  one 
subproblem.  Others  can  be  identified  when  the  life  cycle  management  process  is 
studied  more  closely.  The  background  and  details  of  the  DoD  life  cycle  process 
are  given  in  reference  9. 

This  research  focuses  on  the  testing  aspects  of  software  development  and 
assumes  that  requirements  are  clearly  stated  and  defined  for  the  computer  pro¬ 
grams  being  developed.  The  approach  taken  was  to  define  a  technical  objective 
related  to  software  test  driver  development  and  reliability  for  readiness.  The 
objective  of  this  research  was  to  develop  standardized  methodologies  for  the 
design  and  use  of  software  test  drivers  which  make  it  possible  to  assure  a  uni¬ 
form  level  of  confidence  for  the  software  quality  beyond  the  critical  function 
stress  tests. 

The  following  section  presents  research  which  classifies  various  test  tech¬ 
niques  and  provides  background  for  a  basic  understanding  of  the  current  litera¬ 
ture  available. 


Purpose 


A  test  driver  is  a  simulated  program  which  passes  data  to  the  module  under 
test  and  receives  data  which  the  module  has  processed.  Test  drivers  are  fre¬ 
quently  used  to  test  software  and  demonstrate  that  the  developed  software  con¬ 
forms  to  system  requirements.  Investigation  and  development  of  a  standardized 
methodology  for  the  design  of  software  test  drivers  is  needed  to  assure  a  uniform 
level  of  confidence  for  software  quality.  This  research  project  was  initiated  to 
explore  existing  methodologies  and  evaluate  their  strengths  and  weaknesses.  The 
technical  goal  is,  therefore,  to  discover  an  optimum  system  testing  procedure 
whereby  a  standardized  testing  baseline  can  be  established.  If  successful,  the 
Army  and  the  Department  of  Defense  will  be  able  to  use  this  standard  methodology 
in  the  development  of  reliable  software  for  battlefield  automated  systems,  com¬ 
bining  academic  expertise  with  practical  structured  testing  methods. 

The  program  plan  consisted  of  the  following  key  tasks  (table  1)  which  were 
accomplished  during  phase  1: 

1.  Prepare  plan  for  phase  I 

2.  Initiate  prebidders  conference  for  consultant  selection 

3.  Review  meetings  with  expert  consultant 

4.  Conduct  literature  search  on  test  drivers 

5.  Identify  experts  in  software  testing  arena 

6.  Identify  significant  parameters 

7.  Review  and  finalize  current  models 


8.  Determine  applicability  of  models 

9.  Determine  limitations  of  models 

10.  Analyze  models  and  prepare  recommendations 

11.  Prepare  technical  report 

Phase  1  identifies  significant  testing  models  that  were  evaluated  to  opera¬ 
tionalize  a  standard  approach  in  the  development  of  test  drivers  to  assure  high 
quality  software.  Phase  2  will  address  the  implementation  of  phase  1  and  iden¬ 
tify  a  transition  team  of  researchers  who  will  pursue  further  "hands-on"  develop¬ 
ment  work. 


CLASSIFICATION  OF  TEST  TECHNIQUES 


Background 

Over  the  last  decade,  many  different  test  techniques  have  emerged.  At  pres¬ 
ent,  there  is  no  superior  technique;  however,  most  of  the  commercial  and  experi¬ 
mental  test  programs  are  based  upon  one  or  two  approaches  to  testing.  Therefore, 
even  if  these  early  commercial  test  products  are  imperfect,  the  approaches  upon 
which  they  are  built  must  be  classified  as  the  most  advanced  in  a  practical 
sense.  It  is  realized  that  these  approaches  must  have  potential  if  developers 
have  invested  significant  funds.  Some  practitioners  have  used  these  tools  with 
success,  and  many  others  are  experimenting  with  these  programs. 

Some  recent  progress  has  been  made  in  characterizing  and  defining  the  vari¬ 
ous  test  approaches  (refs  4  and  6). 


Specification  Languages 


Since  many  of  the  errors  which  occur  in  a  project  are  due  to  ambiguous  or 
imprecise  specification  of  the  problem,  various  researchers  have  been  working  on 
languages  which  would  make  the  specification  process  more  precise.  Most  of  these 
techniques  combine  a  formal  specification  langage  with  some  aspects  of  an  inter¬ 
active  definition  mode  on  a  computer  and  are  interspersed  with  English  language, 
comments,  and  some  analysis  capability. 

The  analysis  generally  attempts  to  check  for  completeness,  consistency,  and 
ambiguity  of  the  definitions,  constraints,  and  statements  in  the  specification 
language.  Some  of  the  best  developed  techniques  are  the  A-7  specification  tech¬ 
nique  (ref  10)  and  the  problem  statement  language  and  analyzer  (PSL/PSA)  (ref 
11).  Most  large  software  developers  now  use  some  type  of  pseudo-code  based  pro¬ 
gram  design  language  (PDL). 


Comprehensive  Software  Development  Methodologies 


Other  researchers  have  approached  the  problem  of  errors  which  occur  between 
the  problem  description  and  the  coding  phases.  This  is  done  by  providing  a  com¬ 
prehensive  tool  which  allows  the  researcher  to  go  from  the  basic  concept  of  the 
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problem  to  a  specification  and  then  automatically  generate  code  based  upon  the 
rigorously  defined  specification.  Such  techniques  are  best  described  by  refer¬ 
ence  to  particular  examples  of  methodologies,  e.g.,  the  Higher  Order  Systems 
(HOS)  methodology  (refs  12,  13),  and  the  software  requirement  engineering  method¬ 
ology  (SREM)  (ref  14). 


Performance  Testing 


The  term  performance  testing  normally  applies  to  the  definition,  modeling, 
and  measurement  of  the  computer  or  computer  system.  Typical  measures  that  are 
used  are  millions  of  instructions  per  second  (MIPS),  millions  of  floating  point 
operations  per  second  (MFLOPS),  and  jobs  processed  per  minute.  Analysis  of  this 
last  measure  includes  the  load  on  a  multiprocessing  or  time-sharing  system,  the 
task  assignment,  and  queuing  aspects  of  the  operating  system. 


Code  Walkthroughs 


A  code  walkthrough  is  an  informal  design  review.  The  participants  are  a 
lead  programmer,  the  designer  of  the  code,  and  possibly  one  or  more  pro¬ 
grammers.  The  procedure  involves  the  program  designer's  explaining  his  design, 
the  underlying  algorithms,  and  the  code  to  the  remainder  of  the  group.  Some 
combination  of  flow  charts,  pseudo  code,  or  other  design  representation,  along 
with  a  blackboard  presentation  or  overhead  transparencies  are  used  to  explain  the 
code  design.  The  main  advantage  of  this  technique  is  that  several  people  inter¬ 
act,  and  the  process  minimizes  the  great  tedium  and  boredom  of  code  reading. 


Classification  of  Errors  and  Test  Types 

From  the  literature  reviewed,  various  categories  of  errors  have  been  identi¬ 
fied.  These  errors  can  be  type  classified  into  the  following  areas: 

1  .  Logic 

2.  Documentation  errors 

a .  Code 

b.  Specifications 

c.  Unit  development  folders 

3.  Overload  or  overflow  of  range 

4.  Timing  errors 


5.  Throughput 


6.  Recovery  errors 

7.  Support  software  errors  (test  program  sets) 

8 .  Hardware  errors 


9.  Specifications  and  requirements 

10.  Compilation  errors 

11.  Interface  errors  (software) 

12.  Data  or  data  base  errors 

13.  Incorrect  reporting 

Errors  can  be  identified  by  various  methodologies  and  testing  strategies  can 
be  developed.  Each  testing  method  has  advantages  and  disadvantages  for  various 
types  of  errors  discovered.  Universal  test  methods  and  procedures  may  be  very 
difficult  to  standardize.  An  effective  approach  to  testing  may  involve  a  feed¬ 
back  process  where  various  strategies  of  testing  are  performed,  results  analyzed 
and  then  modified  based  on  the  particular  errors  that  are  discovered.  This  pro¬ 
cess  of  interactive  testing,  analyzing,  and  retesting  based  on  types  of  errors 
detected,  may  be  the  most  cost  effective  procedure  to  investigate  in  test  driver 
formulation . 

Classes  of  testing  are  associated  with  various  errors.  Classification  can 
be  segmented  into  the  following  types: 

1.  Type  0  -  All  instructions  in  code  executed  at  least  once  (check 
list) 

2.  Type  1  -  All  paths  force-executed  at  least  once  (simulated  100% 
coverage) 

3.  Type  1 .5  -  All  paths  force-executed,  some  naturally  executed 

4.  Type  2  -  All  paths  naturally  executed  at  least  once  (path  coverage 
100%) 

5.  Type  3  -  All  paths  naturally  executed  for  all  values  of  input  param¬ 
eters  (exhaustive  test) 

6.  Type  4  -  All  paths  naturally  executed  for  all  values  of  input 
parameters,  all  sequences  of  inputs,  and  all  combinations  of  initial 
conditions  (exhaustive  test  for  multiprocessing,  multiprogramming, 
and  real  time  systems  with  nonfixed  input  sequence) 

A  more  complete  discussion  of  these  classifications  of  testing  is  given  in 
reference  15. 
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Static  and  Dynamic  Testing 


At  the  present  time  there  are  several  testing  methodologies ,  one  of  which  is 
static  testing.  A  static  analysis  is  defined  as  a  direct  analysis  of  the  form 
and  structure  of  a  product  without  executing  the  product  (ref  16).  This  may 
include  code  inspection  techniques,  code  walkthroughs,  or  symbolic  analysis. 
Natural  Inputs  are  not  applied  to  the  program  since  this  form  of  testing  does  not 
involve  execution  or  run-time.  This  strategy  in  testing  may  be  viewed  as  an 
extentlon  of  the  compilation  process  (ref  4).  Static  testing  can  also  be  thought 
of  as  making  certain  allegations  about  the  program  and  then  proving  these  allega¬ 
tions.  Once  the  discovered  errors  are  corrected  and  static  testing  Is  completed, 
the  next  phase  may  include  dynamic  testing. 

Dynamic  testing  involves  execution  or  simulation  of  a  development  phase 
product.  It  detects  errors  by  analyzing  the  response  of  a  product  to  sets  of 
input  data  (ref  16).  In  addition,  dynamic  testing  is  used  for  software  qualifi¬ 
cation  in  the  production  phase  of  the  life  cycle.  Dynamic  testing  is  described 
in  terms  of  the  following  four  elements  (ref  4): 

1.  Set  an  objective  for  the  test  which  will  usually  show  that  the  pro¬ 
gram  is  running  according  to  the  design  requirements. 

2.  Set  up  appropriate  input  data  for  testing  objectives. 

3.  Review  output  and  results  according  to  test  objectives. 

4.  Develop  measurements  which  quantify  results  and  knowledge  obtained 
about  the  program  by  executing  the  program. 

By  executing  the  program  in  a  controlled  environment,  dynamic  testing  is 
used  to  demonstrate  that  errors  are  detected  and  eliminated.  It  assures  that  the 
program  operates  in  the  required  fashion  to  detect  unwanted  or  unnecessary  func- 
t ions . 


Set  Theory  Analyses 


Another  testing  methodology  uses  the  concepts  of  set  theory.  There  are 
generally  two  “solution"  spaces,  one  for  input  and  the  other  for  output.  The 
methodology  involves  mapping  each  element  of  the  input  space  into  a  corresponding 
output  space.  The  concept  is  that  each  input  and  output  should  be  accounted  for, 
and  there  should  not  be  one  without  the  other.  If  such  an  input  or  output  is 
found  to  exist,  it  means  that  either  a  mistake  in  program  logic  exists  (unneces¬ 
sary  data),  or  possibly  an  “added  feature"  is  present.  The  solution  spaces  can 
also  be  broken  down  into  corresponding  domains,  subsets,  or  subspaces. 

This  testing  methodology  is  described  in  reference  17  in  detail.  Briefly, 
the  subset  or  subspace  is  usually  one  of  the  program's  paths.  By  using  this 
methodology,  certain  classes  of  errors  and  programs  can  be  defined.  Errors  in¬ 
clude  domain  errors,  computation  errors,  and  subcase  errors.  Programs  may  be 
reliable,  almost  reliable,  feasible,  or  infeasible.  (For  further  details  on  the 
methodology,  see  reference  17.) 


A  methodology  called  functional  program  testing  is  described  in  reference  18 
and  is  similar  to  the  one  just  mentioned.  According  to  this  methodology,  a  pro¬ 
gram  is  viewed  as  a  collection  of  functions  each  having  input  values  for  its 
variables  and  corresponding  output  values.  The  domains  for  the  input  and  output 
are  formally  specified.  The  data  contained  within  each  of  these  domains  corre¬ 
sponds  to  various  functions  and  they  may  have  one  or  more  important  properties. 
Test  data  are  determined  by  considering  these  properties  and  reviewing  the  de¬ 
fined  domains.  (For  additional  depth  on  functional  program  testing,  see  refer¬ 
ence  18.) 


Graph  Theory 


Graph  theory  describes  program  logic  by  mapping  data  flow  and  control  by 
means  of  flow  charts  and  data  flow  diagrams.  The  flow  chart  is  used  to  define 
program  paths  and  derive  test  cases  to  test  these  paths.  An  automated  method 
which  develops  an  optimal  set  of  test  cases  where  all  branches  of  the  source  code 
are  exhaustively  tested  is  described  in  reference  19.  This  process  uses  a  mini¬ 
mum  set  of  paths  which  cover  all  logical  branches  of  the  module.  The  testing 
process  shows  whether  or  not  the  program  successfully  performs  its  intended  func¬ 
tion  in  an  acceptable  fashion  (e.g.,  efficiently,  timely)  for  every  combination 
of  input  variable  values  and  every  conceivable  program  path  given. 

The  graph  theory  method  consists  of  the  following  procedures:  The  source 
code  is  analyzed  for  its  syntax.  Certain  branch  conditions  are  identified  as 
being  Incompatible.  These  incompatibilities  may  render  a  path  unexecutable  and 
they  can  be  eliminated.  However,  some  branch  conditions  may  not  be  detected. 
This  may  require  that  the  user  input  some  information.  The  user  then  generates 
test  data  which  will  exercise  all  paths .  A  path  which  contains  the  maximum  num¬ 
ber  of  branches  not  yet  exercised  is  identified.  A  test  case  to  drive  this  path 
is  generated  in  order  to  exercise  the  path.  The  process  is  repeated  until  all  of 
the  branches  are  exercised.  Additional  information  on  this  methodology  can  be 
obtained  from  reference  19. 


Structured  Testing 


The  test  methodology  of  structured  testing  relates  to  the  concepts  of  graph 
theory.  McCabe  &  Associates  (ref  20)  have  done  extensive  work  in  this  field.  By 
determining  the  data  flow  diagram  for  a  program  under  study,  McCabe  has  developed 
a  metric  which  gives  an  indication  of  the  complexity  of  the  program.  The  value 
of  the  complexity  metric  gives  the  minimum  number  of  distinct  paths  (basis  paths) 
which  must  be  tested  to  assure  software  reliability.  If  the  module  or  procedure 
has  a  cycloraatic  complexity  greater  than  ten,  then  the  module  is  more  likely  to 
have  errors.  Furthermore,  the  module  will  be  more  difficult  to  understand,  test, 
and  debug.  In  addition,  inherent  timing  problems  may  result,  and  the  module  will 
be  difficult  to  maintain.  To  assure  ease  of  understanding  and  testability,  mod¬ 
ules  should  be  broken  down  into  simpler  components,  each  having  a  complexity  of 
ten  or  less. 


Modules  or  redesigned  modules  should  be: 

1 .  Testable  In  the  sense  that  the  testing  effort  can  be  managed  prop¬ 
erly. 

2.  Comprehensible  so  that  the  user  can  easily  read  and  understand  what 
Is  being  done. 

3.  Definable  so  that  It  may  be  used  In  another  system. 

4.  Maintainable  to  facilitate  proper  management. 


EVALUATION  OF  TEST  APPROACHES 


Comparison  of  Features 


The  various  test  techniques  which  were  Introduced  cannot  be  applied  to  all 
phases  of  software  life  cycle  development  with  equal  ease  or  validity.  A  compar¬ 
ison  of  the  effectiveness  of  these  various  approaches  and  features  appears  In 
table  2. 

The  following  main  points  from  the  table  should  be  considered: 

1 .  Many  of  the  techniques  involve  a  human  analyst  Interacting  with 
other  key  personnel  during  the  life  cycle  management  process  or  examining  some 
stage  of  the  development  cycle.  The  elimination  of  the  human  factor  element  from 
the  testing  process  cannot  be  envisioned.  However,  the  most  desired  objective  is 
to  develop  an  automated  tool  or  technique  which  will  provide  an  additional  means 
of  finding  errors  and  reducing  the  effort  and  the  large  number  of  manhours  in¬ 
volved  in  testing  a  large  program  in  battlefield  automated  systems. 

2.  The  first  priority  would  be  to  reduce  further  work  on  code  reading, 
code  walkthroughs,  design  reviews,  program  proofs,  set  theory  analysis,  and  func¬ 
tional  program  testing  since  these  areas  are  too  complex  and  impractical  for 
automation. 

3.  Much  work  Is  already  in  process  on  specif ication  languages  and  com¬ 
prehensive  methodologies.  At  a  later  stage,  these  avenues  will  be  explored  fur¬ 
ther  based  upon  state-of-the-art  progress. 

4.  Since  structured  testing  and  test  path  approaches  have  been  auto¬ 
mated,  they  appear  to  be  the  most  promising  approaches  for  phase  2.  Some  of  the 
features  of  symbolic  testing  have  been  incorporated  In  the  programs  which  exer¬ 
cise  test  paths.  Current  trends  in  software  development  packages  indicate  that 
programming  languages  such  as  PASCAL  and  Ada  are  being  used  for  current  Depart¬ 
ment  of  Defense  projects.  These  structured  programming  languages  and  techniques 
are  forcing  contractors  to  develop  structured  testing  methodologies  in  order  to 
qualify  their  software  packages. 


Based  on  these  ideas,  the  following  sections  present  the  linkage  between 
structured  testing  and  automated  tools.  Structured  testing  is  further  elaborated 
upon  with  respect  to  the  cyclomatic  complexity  metric  developed  by  McCabe  & 
Associates,  Inc.  The  automated  tool  considered  for  further  evaluation  for  phase 
2  is  the  Test  Coverage  Analysis  Tool  (TCAT)  developed  by  Software  Research  Asso¬ 
ciates. 


Control  Graphs 


Some  of  the  most  promising  software  test  tools  developed  to  date  are  related 
to  path  testing  of  the  software.  The  simplest  way  to  define  software  paths  is  to 
refer  to  the  control  graph  for  the  software.  An  approximate  notion  of  the  con¬ 
trol  graph  can  be  quickly  obtained  if  the  flow  chart  for  the  software  is  consid¬ 
ered.  If  each  start-stop  oval,  processing  box,  input-output  rhombus,  and  deci¬ 
sion  diamond  are  replaced  by  a  small  circle,  then  the  nodes  of  the  graph  are 
defined.  The  flow  lines  in  the  flow  chart  that  form  the  branches  of  the  graph 
are  retained.  (In  mathematical  terms,  the  nodes  and  branches  are  called  vertices 
and  arcs.)  In  general  there  are  two  types  of  graphs,  directed  (digraphs)  and 
nondlrected.  In  a  directed  graph,  the  branches  can  only  be  traversed  in  the 
indicated  arrow  direction.  In  an  undirected  graph  the  branches  can  be  traversed 
in  both  directions.  If  computer  programs  are  modeled,  directed  graphs  must  be 
used . 

Computer  programs  are  initially  considered  without  loops,  therefore  the  flow 
charts  do  not  have  loops,  nor  do  the  associated  control  graph.  If  only  struc¬ 
tured  programs  are  considered,  then  loopless  programs  contain  only  SEQUENCE  and 
IF  THEN  ELSE  control  structures  and  do  not  contain  DO  WHILE  control  structures. 
Such  a  control  graph  contains  only  two-way  brances  at  each  IF  THEN  ELSE  struc¬ 
ture,  which  creates  the  paths.  A  path  is  a  unique  sequence  of  branches  which 
connects  the  start  and  stop  nodes  in  such  a  graph.  A  path  test  would  generate 
test  data  to  drive  the  program  down  all  or  some  subset  of  all  the  graph  paths . 

If  program  loops  (add  DO  WHILE  or  DO  UNTIL)  control  structures  in  the  code 
are  allowed,  the  flow  chart  and  the  associated  control  graph  will  contain 
loops.  Loops  make  the  definition  of  paths  more  difficult,  since  every  additional 
“trip  around  a  loop"  will  create  a  new  path.  To  avoid  this  problem,  a  loop  in  a 
program  Is  defined  as  generating  two  paths:  one  is  when  the  WHILE  condition  is 
initially  false,  and  the  loop  is  not  executed;  the  other  is  when  the  loop  is 
executed  only  the  first  time  (ref  6). 


Cyclomatic  Complexity 


A  complexity  measure  for  the  control  structure  of  a  computer  program  by 
adapting  graph  theory  was  devised  by  McCabe  (ref  20).  This  is  known  as  the  cy¬ 
clomatic  complexity  of  the  graph  and  is  related  to  the  number  of  "independent" 
loops  in  the  graph.  The  complexity  can  be  calculated  in  several  fashions,  one  of 
which  is  by  adding  unity  to  the  difference  between  the  number  of  branches  and  the 
number  of  nodes  of  the  graph  (refs  6  and  20). 


McCabe  further  relates  the  cyclomat lc  complexity  to  the  minimum  number  of 
test  paths  which  are  needed  to  cover  (pass  through)  all  the  branches  in  the  graph 
of  the  program.  This  technique  can  be  implemented  to  generate  the  minimum  set  of 
tests  for  exercising  the  program  and  assuring  software  reliability  (refs  6  and 
20). 


Test  Coverage  Analysis  Tool 


A  test  tool  called  Test  Coverage  Analysis  Tool  (TCAT)  has  been  developed  and 
marketed  by  Edward  Miller  of  Software  Research  Associates .2  This  tool  is  based 
on  path  testing  and  generates  a  set  of  test  conditions  which  drives  the  program 
down  its  various  paths.  Miller  claims  that  although  the  tool  will  not  generate 
test  conditions  for  all  the  program  paths ,  it  will  find  and  test  up  to  85%  of  the 
paths  (test  coverage  of  0.85)  in  examples  which  he  has  investigated.  It  also 
provides  a  static  analysis  of  the  program  along  with  the  dynamic  path  testing. 

This  tool  is  available  in  several  versions  which  run  on  a  few  different 
computers.  TCAT  appears  to  be  one  of  the  more  advanced  and  flexible  tools  which 
are  commercially  available.  Because  of  these  strengths,  it  is  recommended  that 
one  or  more  versions  of  TCAT  be  acquired  for  phase  2  of  this  research  project, 
and  that  its  capabilities  and  limitations  be  thoroughly  explored. 


Cycloraatic  Complexity  Measure  in  Testing 


McCabe's  cyclomatic  complexity  measure  can  be  used  to  supplement  a  path 
testing  program  such  as  TCAT.  The  cyclomatic  complexity  can  be  computed  and  used 
as  a  lower  bound  on  the  number  of  path  tests  necessary  for  testing  the  program  in 
question.  It  is  advantageous  to  know  if  there  are  tens,  hundreds,  or  thousands 
of  test  paths  necessary  before  TCAT  or  similar  test  programs  are  used. 

Another  use  of  the  cyclomatic  complexity  measure  is  in  code  inspection.  A 
large  number  of  programs  and  program  modules  can  be  submitted  to  a  cyclomatic 
complexity  analysis  program  and  the  various  programs  ranked  with  respect  to  their 
cyclomatic  complexity.  Rewriting  or  further  study  of  the  most  complex  programs 
in  the  group  would  then  be  considered. 

Many  private  concerns  are  using  automated  tools  to  aid  in  their  software 
development  and  analysis.  For  example.  Ultrasystems  from  Irvine,  California,  has 
a  number  of  automated  tools  in  operation.  One  tool  is  a  JOVIAL  Code  Analysis 
Program  (JCAP).  JCAP  is  a  static  analysis  tool  which  provides  information  on 
standards  auditing,  structure  analysis,  data  usage,  flow  analysis,  and  complexity 
metrics.  Similar  to  JCAP,  an  Assembly  Code  Analysis  Program  (ACAP)  is  available 
to  use  for  static  analysis  of  assembly  code. 
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Correspondence  between  Miller  and  authors. 
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Another  automated  tool  which  Is  still  In  development  by  Ultrasystems  Is 
Multi-Language  Static  Analysis  Tool  (MSAT).  The  MSAT  program  structure  analysis 
provides  Information  on  procedure  structure  call  charts,  intramodule  charts,  and 
nesting  level  of  procedures.  The  data  flow  analysis  examines  module  coupling 
analysis,  global  cross  reference,  constant  usage,  data  usage,  input/output  re¬ 
ports,  intramodule  input/output  reports,  module  strength  analysis,  and  program 
stability  analysis.  The  MSAT  auditing  tool  investigates  standards  compliance. 
It  provides  complexity  metrics  and  error  analysis  which  do  global  data  checks, 
constant  checks,  and  provide  reports  on  system  completeness,  procedure  complete¬ 
ness,  and  procedure  usage  errors.  The  MSAT  tool  provides  general  system  informa¬ 
tion  on  listings,  procedure  maps,  assembly  percentages,  comment  percentages,  and 
abstracts.  It  is  being  developed  to  examine  software  changes.  Code  change  anal¬ 
ysis  and  structure  change  analysis  will  be  provided  in  the  MSAT  tool.  According 
to  Ultrasystems,  this  tool  is  still  under  development  but  may  be  completed  within 
a  year. 

Cyclomatlc  complexity  analysis  is  also  used  in  business  application  pro¬ 
grams.  AT&T  in  Warrenville,  Illinois  has  expanded  their  software  quality  assur¬ 
ance  group  in  order  to  provide  feedback  to  designers  of  software  programs  on 
errors  detected  through  cyclomatlc  complexity  analysis  and  other  automated  test 
tools. 


CHARACTERISTICS  OF  TOOL 


After  reviewing  additional  testing  methodologies  and  techniques,  general 
characteristics  for  a  potential  generic  automated  tool  were  formulated.  Attri¬ 
butes  to  consider  are: 

1.  User  requirements  and  wants 

2.  Tool  objectives 

3.  Tool  usage 

4.  Tool  languages 

5.  Primary  features 

6.  Cost 

As  a  pilot  study,  a  simple  problem  was  analyzed  to  determine  characteristics 
of  an  automated  tool.  This  analysis  was  performed  manually,  keeping  in  mind  the 
process  involved  and  how  the  computer/tool  could  be  used  to  perform  the  same 
task.  The  steps  followed  in  this  procedure  consisted  of  developing  a  flow  chart, 
enumerating  the  number  of  paths,  labeling  the  paths  by  some  means  to  distinguish 
one  from  another,  and  generating  data  to  drive  a  certain  path. 


User  Interface 


In  developing  a  generic  tool,  the  question  of  user  interface  comes  to 
mind.  The  tool  could  be  adaptable  to  a  number  of  program  languages.  This  would 
depend  on  the  resources  and/or  hardware  system  capabilities  available  to  the 
user.  The  coordination  between  user  requirements  and  company  policies  needs  to 
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be  considered.  To  facilitate  the  use  and  management  of  the  tool,  a  user's  manual 
should  be  provided  with  typical  examples  including  a  complete  record  of  the  test 
procedures.  Preferably,  there  should  be  a  concise  version  of  the  user's  manual 
on  an  interactive  computer  terminal  which  includes  a  user  friendly  HELP  program 
menu. 


User  Friendly 


One  practical  aspect  of  the  tool  would  be  to  make  it  user  friendly.  User 
menus  and  prompts  can  guide  the  user  through  tool  usage  with  a  simplified  set  of 
Instructions.  Having  the  user  interact  with  the  tool  would  make  it  convenient 
and  easy  to  understand  what  has  been  done  or  what  will  be  done.  It  can  also  give 
some  indication  as  what  to  do  next.  This  feedback  process  allows  software  to  be 
examined  and  changed  easily.  For  example,  if  the  program  is  too  complex  and  the 
number  of  paths  too  large  to  test,  the  tool  could  ask  the  user  to  pick  a  particu¬ 
lar  path  he  might  want  to  test.  It  might  also  ask  the  user  if  a  retest  were 
needed.  This  potential  retest  option  would  increase  user  confidence  in  tool 
reliability  through  the  retest  enhancement.  The  tool  could  have  the  ability  to 
give  feedback  on  critical  paths  that  should  be  taken  next  and  what  input  should 
be  used  to  exercise  the  path.  In  addition,  an  automatic  data  generator  could 
drive  the  desired  path. 


Output  and  Graphics 


Probably  one  of  the  most  important  features  of  the  tool  would  be  output.  It 
may  discuss  the  number  of  paths  In  the  program,  the  labeling  of  these  paths,  the 
percentage  of  paths  tested,  how  many  times  each  of  the  paths  was  tested,  or  the 
relative  complexity  of  the  program.  The  output  should  be  in  a  format  which  is 
easy  to  interpret  and  comprehend.  The  use  of  graphics  would  simplify  this  pro¬ 
cess.  A  tree  diagram  or  flow  chart  could  be  used  to  illustrate  program  logic  or 
data  flow.  This  could  easily  show  the  various  paths  of  the  program.  By  number¬ 
ing  or  lettering  the  nodes  and/or  decision  points,  the  various  paths  can  be  de¬ 
fined.  When  a  path  is  executed,  the  output  could  consist  of  the  numbers  or  let¬ 
ters  to  delineate  the  program  path.  If  graphics  are  involved,  the  particular 
path  could  be  highlighted  by  means  of  colors ,  dashed  lines ,  or  some  other  conven¬ 
ient  method.  A  bar  graph  may  be  used  to  show  the  various  paths  and  the  relative 
frequency  or  percentage  of  times  each  path  was  tested  and/or  executed. 


Software  Research  Associates  has  several  software  testing  support  tools. 
TCAT^  illustrates  the  tool  characteristics  mentioned  above.  It  is  UNIX-based  and 
available  in  COBOL,  RATFOR,  SRTRAN,  and  BASIC.  The  tool  analyzes  the  program  and 
defines  modules  and  segments.  The  tool  tests  approximately  85%  of  the  paths  and 
also  tests  the  inputs  and  outputs  of  the  program.  One  of  the  outputs  consists  of 
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listing  each  nodule  and  identifying  the  segments,  if  any,  which  are  not  exer¬ 
cised.  The  TCAT  Summary  Test  Coverage  Report  lists  the  following  critical  items: 

1 .  Module  name 

2.  Number  of  segments 

3.  Number  of  invocations 

4 .  Number  of  segments  exercised 

5.  Percentage  covered 

On  the  TCAT  Histogram  Report ,  the  segment  number  and  the  number  of  execu¬ 
tions  of  the  segment  are  given  and  displayed  in  a  histogram  format .  Many  other 
leading  companies  have  similar  tools  which  are  commercially  available;  however, 
researchers  need  to  be  very  selective  in  purchasing  automated  tools. 

CONCLUSIONS 


The  objective  of  phase  1  was  to  investigate  the  developing  standardized 
methodologies  for  the  design  of  software  test  drivers.  Test  drivers  are  neces¬ 
sary  to  assure  a  uniform  level  of  confidence  for  the  software  quality  beyond  the 
critical  function  stress  tests.  These  standardized  test  drivers  are  being  con¬ 
sidered  for  use  in  battlefield  automated  systems  within  the  Department  of 
Defense. 

A  literature  survey  and  personal  interviews  with  leading  software  devel¬ 
opers,  such  as  AT&T  and  Ultrasystems,  were  conducted.  Static  and  dynamic  testing 
methodologies  were  investigated  with  respect  to  the  advantages  and  disadvan¬ 
tages.  As  a  result,  a  standardized  automated  tool  for  each  application  seems 
highly  unlikely.  Therefore,  a  generic  strategy  for  developing  a  standardized 
test  driver  needs  further  investigation. 


RECOMMENDATIONS 


It  is  recommended  that  the  most  promising  test  driver  methodologies  be  exam¬ 
ined  in  phase  2.  These  include  the  Cycloraatic  Complexity  Measure  of  McCabe  & 
Associates,  Inc.  and  the  Test  Coverage  Analysis  Tool  (TCAT)  of  Software  Research 
Associates . 

The  long  term  objective  is  to  develop  a  standard  methodology  that  can  be 
automated  as  a  generic  tool  (test  driver)  to  be  used  in  Army  and  Department  of 
Defense  Software  Quality  Assurance  Centers  to  increase  software  reliability. 
Strategies  for  tailored  automated  test  drivers  will  be  generated  to  assure  in¬ 
creased  confidence  in  software  development  throughout  the  life  cycle  of  the  bat¬ 
tlefield  automated  system.  To  achieve  this  objective,  continued  research  is 
proposed  to  supplement  this  effort.  The  planning  for  the  next  phase  of  the  re¬ 
search  is  broken  down  into  the  following  steps: 

1.  Investigate  tools  that  are  available,  purchase  the  tools,  and  verify 
that  the  tools  actually  perform  their  intended  function.  In  essence,  check  the 


tools  for  content  validity.  As  a  result  of  this  research,  several  tools  can  be 
validated  and  used.  As  previously  mentioned.  Software  Research  Associates  have 
several  automated  tools.  Other  industries  which  have  similar  tools  include 
Boeing,  McDonnell  Douglas,  IBM,  Bell  Laboratories,  and  Softool.  Within  the  gov¬ 
ernment,  the  Federal  Software  Testing  Center  in  Washington,  DC  has  additional 
software  tools  available;  however,  not  all  of  these  tools  are  commercially  avail¬ 
able. 

2.  Study  the  fundamentals  of  the  tool.  This  would  consist  of  determin¬ 
ing  how  the  logic  diagram,  flow  chart,  or  paths  are  defined.  Basic  factors  such 
as  what  inputs  are  necessary  to  exercise  a  particular  path  and  how  to  interpret 
the  output  are  important.  The  result  is  a  basic  understanding  of  how  the  tool 
operates  and  experiencing  its  limitations. 

3.  Introduce  a  representative  set  of  errors  into  the  program.  Since  it 
is  difficult  to  seed  errors,  an  obsolete  program  with  known  errors  can  be  used  to 
check  the  validity  of  the  tool.  This  effort  would  build  on  steps  one  and  two. 
Knowledge  and  insight  gained  would  help  users  to  understand  the  fundamentals  of 
the  tool.  Additional  significant  findings  are  then  documented  in  the  updated 
user’s  manual. 

4.  Summarize  the  research  by  modifying  and  consolidating  the  lessons 
learned  from  selected  tools  and  create  a  prototype  test  driver. 

5.  Revise  the  user's  manual  for  the  tool.  It  is  assumed  that  the  tools 
are  written  by  experts  for  use  by  experts.  A  user's  manual  would  be  developed  in 
order  that  the  user  could  fully  understand  the  basic  ideas,  applications,  and 
practicality  of  the  tool  in  Army  and  DoD  environments. 
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Table  1 .  Test  driver  R&D  program  schedule  and  milestones 


1983 


TASK 


MAY  JUN  I  JDL  AUG  .  SEP  |  OCT  NOV  DEC 
I _  I 


1.  PREPARE  PLAN  FOR  PHASE  1 


2.  PREBIDDERS  CONSULTANT 
CONFERENCE 

3.  REVIEW  MEETING  WITH 
CONSULTANTS 

4.  CONDUCT  LITERATURE  SEARCH 

5.  INTERVIEW  EXPERTS 

6.  IDENTIFY  SIGNIFICANT 
PARAMETERS 

7.  REVIEW  AND  FINALIZE 
MODELS 

8.  DETERMINE  APPLICABILITY 
OF  MODELS 

9.  DETERMINE  LIMITATIONS  OF 
MODELS 

10.  ANALYZE  AND  RECOMMEND 
MODELS 

11.  PREPARE  FINAL  REPORT 


LEGEND!  A  START 

A  COMPLETE 
0  PLANNED  SLIP 
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GLOSSARY 


ACAP:  Assembly  Code  Analysis  Program. 

ACCEPTANCE  TESTING:  The  validation  of  the  system  or  program  to  the  user  require¬ 
ments. 

BACKTRACKING:  Examining  the  error  symptoms  to  see  where  they  were  first  noticed 
and  then  backstepping  in  the  program  flow  of  control  to  a  point  where  the  symp¬ 
toms  have  disappeared. 

BIG  BANG  TESTING:  When  all  modules  have  been  individually  coded  and  are  sub¬ 
mitted  to  integration  testing  without  prior  unit  testing. 

BUILD:  A  version  of  a  program  incorporating  a  subset  of  the  required  features. 

Software  goes  through  many  builds  as  it  evolves  in  a  project. 

CERTIFICATION:  An  authorative  endorsement  of  the  program.  Testing  for  certifi¬ 

cation  must  be  done  against  some  predefined  standard. 

CLASSIC  SOFTWARE  DEVELOPMENT:  Development  which  has  proceeded  bottom-up  in 

design,  coding,  and  testing. 

CODE  READING:  The  reading  of  a  program,  by  another  programmer  other  than  the 
designer,  with  the  purpose  of  finding  errors. 

CONFIGURATION  CONTROL:  The  strict  control  of  the  source  code  of  a  program  under 
development.  The  controller,  called  the  configuration  manager,  keeps  the  offi¬ 
cial  copy  of  the  program.  Changes  or  corrections  can  only  be  made  on  written 
request  to  the  configuration  manager  (and  the  configuration  control  board). 

CONFIGURATION  MANAGEMENT:  See  Configuration  Control. 

COVER:  A  set  of  tests  which  cover  (exercise)  all  instructions  in  a  program. 
Might  also  mean  a  set  of  tests  which  exercise  all  control  paths  or  flows  in  a 
program. 

CYCLOMATIC  NUMBER:  For  a  control  graph,  the  cyclomatic  complexity  number  can  be 
calculated  by  taking  the  number  of  edges  (branches),  subtract  the  number  of  ver¬ 
tices  (nodes),  and  add  unity. 

DEBUGGING:  The  activity  of  diagnosing  the  precise  nature  of  a  known  error  and 

then  the  correcting  of  the  error. 

DEDUCTIVE  DEBUGGING:  Process  begins  by  enumerating  all  causes  or  hypotheses 

which  seem  possible,  then  one  by  one,  particular  causes  are  ruled  out  until  a 
single  one  remains  for  validation. 

DESIGN  REVIEWS:  Formal  meetings  held  by  the  project  director  for  his  representa¬ 
tive,  other  professionals,  and  at  times  the  customer's  representative,  to  review 
in  detail  the  progress  of  the  project. 


DESK  CHECKING:  See  Code  Reading. 

DoD:  Department  of  Defense. 

ENHANCEMENT:  The  redesign  of  a  significant  portion  of  the  software  to  provide 

additional.  Improved,  or  changed  functions.  Sometimes  enhancement  Is  wrongly 
classified  as  maintenance. 

EXTERNAL  FUNCTION  TESTING:  The  verification  of  the  external  system  as  stated  In 
the  external  specifications. 

EYE BALLING:  See  Code  Reading. 

FIELD  TESTING:  The  Initial  operation  of  the  actual  hardware  or  software  system 
In  the  field  In  a  test  mode  (limited  or  full  capabilities)  to  ferret  out  as  many 
errors  as  is  feasible . 

HAND  EXECUTION:  See  Code  Reading. 

HOS:  Higher  order  systems. 

INDUCTIVE  DEBUGGING:  Approach  comes  from  the  formulation  of  a  single  working 

hypothesis  based  on  the  data,  on  the  analysis  of  existing  data,  and  on  specially 
collected  data  to  prove  or  disprove  the  working  hypothesis. 

INSTALLATION  TESTING:  The  validation  of  each  particular  installation  of  the 

system  with  the  Intent  of  pointing  out  errors  made  while  installing  the  system. 

INTEGRATION  TESTING:  The  verification  of  the  Interfaces  among  system  parts  imod- 
ules,  components,  and  subsystems).  Tests  are  performed  which  exercise  Interfaces 
among  program  modules. 

JCAP:  JOVIAL  Code  Analysis  Program. 

MAINTENANCE:  The  support  of  operational  software  through  documentation  of  errors 
discovered  In  the  field,  generation  of  work  around  procedures,  correction  of 
errors  (where  appropriate),  and  installation  of  very  minor  changes  and  additions 
to  the  code  (see  Enhancement). 

MFLOPS:  Millions  of  floating  point  operations  per  second. 

MILESTONE:  A  significant  event  in  a  PERT  project  management  chart. 

MIPS:  Millions  of  Instructions  per  second. 

MODERN  SOFTWARE  DEVELOPMENT:  Development  which  has  proceeded  topdown  In  design, 
coding  and  testing. 

MODIFIED  TOP-DOWN  TESTING:  While  integration  testing  of  the  control  structure  Is 
in  progress  the  critical  module  is  being  exercised  with  a  test  driver  program. 


MODULE  TESTING  or  UNIT  TESTING:  The  verification  of  a  single  program  module, 
usually  In  an  Isolated  environment  (i.e.,  isolated  from  all  other  modules). 

MSAT:  Multi-Language  Static  Analysis  Tool. 

PATH:  A  sequence  of  edges  which  when  traversed  in  the  arrow  direction  form  a 

connection  from  the  start  vertex  to  the  stop  vertex. 

PATH  TEST:  The  traversal  of  a  path. 

PDL:  Program  design  language. 

PERT:  Program  Evaluation  and  Review  Technique.  A  diagram  (flow  graph  like) 
which  includes  milestones,  events,  and  paths  between  them.  A  PERT  diagram  is 
used  to  document  and  manage  program  progress. 

PROGRAM  PATH:  A  graph  that  includes  only  a  single  traversal  of  any  loops  which 
are  encountered. 

PROOF:  An  attempt  to  find  errors  in  a  program  without  regard  to  the  program's 

environment. 

PSEUDO-CODE:  A  program  design  representation  technique  composed  of  a  mixture  of 

English  language  statements,  generic  programming  statements,  and  a  few  statements 
specific  to  the  chosen  programming  language.  The  purpose  is  to  produce  a  sort  of 
"pidgen  English”  which  presses  the  main  features  of  the  design  at  a  high  level. 

REGRESSION  TESTING:  Repetition,  in  whole  or  in  part,  of  testing  after  an  error 
has  been  discovered  and  a  correction  has  been  made. 

SANDWICH  TESTING:  Top-down  and  bottom-up  approach  to  testing,  working  from  both 
ends  toward  the  middle.  Used  if  there  is  more  than  one  critical  module  or  if  the 
critical  module  is  totally  persuasive. 

SIMULATION  TESTING:  The  exercise  of  the  software  in  conjunction  with  a  simula¬ 
tion  program  and  often  some  peripheral  hardware  which  replicate  the  real  operat¬ 
ing  environment  as  closely  as  possible.  The  computer  used  to  run  the  software 
and  the  simulation  program  may  be  the  computer  to  be  used  in  the  field  or  a  de¬ 
velopment  computer. 

SOFTWARE  RELIABILITY:  The  probability  that  a  program  will  perform  it's  intended 
task  for  a  certain  period  without  causing  a  system  failure. 

SQA:  Software  Quality  Assurance. 

SREM:  Software  Requirement  Engineering  Methodology. 

STRONGLY  CONNECTED  GRAPH:  A  graph  in  which  each  node  in  the  graph  can  be  reached 
from  any  other  node. 

SYMBOLIC  TESTING:  An  analysis  technique  which  derives  a  symbolic  expression  for 
every  path  of  the  program. 


SYSTEM  INTEGRATION:  The  process  by  which  Individual  modules  are  put  together  to 
realize  major  subsections  and  functions  of  a  program. 

SYSTEM  TESTING:  The  verification  and/or  validation  of  the  system  to  its  initial 
objectives.  System  testing  Is  a  verification  process  when  it  Is  done  in  a  simu¬ 
lated  environment;  it  is  a  validation  process  when  it  is  performed  in  a  live 
environment. 

TCAT:  Test  Coverage  Analysis  Tool. 

TEST  DRIVERS:  A  simulated  program  which  passes  data  to  the  module  under  test  and 
receives  data  which  the  module  has  processed.  Input  data  or  code  used  to  check 
if  the  software  meets  the  requirements  and  does  its  intended  function.  Needed  in 
a  bottom-up  coded  program. 

TESTING:  The  process  of  executing  (or  evaluating)  a  program  (or  part  of  a  pro¬ 
gram)  with  the  intention  or  goal  of  finding  errors.  The  activity  of  finding 
errors. 

TEST  STUBS:  Output  statements  for  each  module  within  the  control  structure  to 
indicate  that  control  has  passed  through  the  module  during  the  tests  of  the  con¬ 
trol  structure.  Needed  in  a  top-down  coded  program. 

VALIDATION:  An  attempt  to  find  errors  by  executing  a  program  in  a  given  real 

environment. 

VERIFICATION:  An  attempt  to  find  errors  by  executing  a  program  in  a  test  or 

simulated  environment. 

WALKTHROUGH:  An  informal  design  review  held  by  a  lead  programmer  and  some  of  his 
staff  to  explore  any  design  or  coding  errors  which  may  exist. 
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